Papers by Karel Joshua Harjono
Towards Reliable Agents: Benchmarking Customized LLM-Based Retrieval-Augmented Generation Frameworks with Deployment Validation (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks for general-purpose RAG systems, such as CRAG, RGB, MultiHop-RAG, and CRUD-RAGG, are limited and lack a benchmark specifically tailored to evaluate frameworks. |
| Approach: | They evaluated OpenAI’s Assistants API versus a RAG assistant built with Langchain and deployed a system based on benchmark insights as a course assistant over a two-year span. |
| Outcome: | The proposed benchmarks show that domain-specific retrieval impacts response accuracy and highlight key challenges in real-world deployment. |